fix(utils.mjs) use the old stats.jenkins.io website (temporarily?) #1890

dduportal · 2024-09-11T18:18:26Z

The plugin site generation fails since jenkins-infra/helpdesk#4265 was applied earlier today.

This PR is a short term fix to use the old stats website, still available through http://old.stats.jenkins.io (and still updated).

These files will have to be served in the new website

cc @lemeurherve @lemeurherveCB @krisstern @MarkEWaite for info

dduportal · 2024-09-11T18:26:53Z

Now it's failing on the Update Center plugins list, most probably due to jenkins-infra/helpdesk#2649

20:22:35  [@jenkins-cd/jenkins-plugin-site]: success Fetching 'https://updates.jenkins.io/plugin-versions.json' - 0.292s
20:22:35  [@jenkins-cd/jenkins-plugin-site]: error gatsby-source-jenkinsplugin: Failed to parse API call -  TypeError: Cannot convert undefined or null to object
20:22:35  [@jenkins-cd/jenkins-plugin-site]:     at Function.values (<anonymous>)
20:22:35  [@jenkins-cd/jenkins-plugin-site]:     at fetchPluginVersions (file:///home/jenkins/agent/workspace/Infra_plugin-site_PR-1890/plugins/gatsby-source-jenkinsplugins/utils.mjs:477:41)
20:22:35  [@jenkins-cd/jenkins-plugin-site]:     at processTicksAndRejections (node:internal/process/task_queues:95:5)
20:22:35  [@jenkins-cd/jenkins-plugin-site]:     at async Promise.all (index 4)
20:22:35  [@jenkins-cd/jenkins-plugin-site]:     at Module.sourceNodes (file:///home/jenkins/agent/workspace/Infra_plugin-site_PR-1890/plugins/gatsby-source-jenkinsplugins/gatsby-node.mjs:19:9)
20:22:35  [@jenkins-cd/jenkins-plugin-site]:     at runAPI (/home/jenkins/agent/workspace/Infra_plugin-site_PR-1890/node_modules/gatsby/src/utils/api-runner-node.js:509:16)

dduportal · 2024-09-11T19:19:52Z

plugin-site/plugins/gatsby-source-jenkinsplugins/utils.mjs

Line 33 in dbf1ff1

if (results.status !== 200) {

=> looks like the Axios HTTP client does not follow redirects 🤦

dduportal · 2024-09-11T19:31:56Z

Ping @timja @halkeye if you can help on this one

timja · 2024-09-11T19:34:04Z

I'm looking

halkeye · 2024-09-11T19:35:44Z

in theory you can do const results = await axios.get(url, { maxRedirects: 10 }); or something to handle the redirects. i can't really look till after work

dduportal · 2024-09-11T19:38:36Z

in theory you can do const results = await axios.get(url, { maxRedirects: 10 }); or something to handle the redirects. i can't really look till after work

One last try for me: 5489791

If it does not work, I don't mind anyone pushing on this branch (or doing another PR) to help as it starts to be late here

dduportal · 2024-09-11T19:42:28Z

in theory you can do const results = await axios.get(url, { maxRedirects: 10 }); or something to handle the redirects. i can't really look till after work

One last try for me: 5489791

If it does not work, I don't mind anyone pushing on this branch (or doing another PR) to help as it starts to be late here

does not work: I might have misunderstood the error. But the file in the new UC is strictly the same as the old UC. Only the redirections are differents.

I'm letting you folks have a look. Many thanks ❤️

timja · 2024-09-11T19:42:38Z

I'm unsure if its redirects axios should handle that automatically, that last change didn't help either, this is easy to run locally

timja · 2024-09-11T20:04:54Z

In node.js repl this is what I get which does not look right:

const axios = require('axios')
const url = "https://updates.jenkins.io/plugin-versions.json"
const results = await axios.get(url);
JSON.stringify(results.data)

result:

{
  "FileInfo": {
    "Path": "/current/plugin-versions.json",
    "Size": 18595440,
    "ModTime": "2024-09-11T19:53:20.9436167Z",
    "Sha1": "432462f8feffc3db439253c2c1ebe618b7346113",
    "Sha256": "9d482e0f77066d7bff52854df41a4ff73fb5967523ddf88fece1020b5bbb1101",
    "Md5": "208824be0422f1009c701562c739b003"
  },
  "IP": "my-ip",
  "ClientInfo": {
    "CountryCode": "GB",
    "ContinentCode": "EU",
    "City": "Brighton",
    "Country": "United Kingdom",
    "Latitude": <redacted>,
    "Longitude": <redacted>,
    "ASName": "My ISP",
    "ASNum": 5089
  },
  "MirrorList": [
    {
      "ID": 0,
      "Name": "fallback0",
      "HttpURL": "https://eastamerica.cloudflare.jenkins.io/",
      "RsyncURL": "",
      "FtpURL": "",
      "SponsorName": "",
      "SponsorURL": "",
      "SponsorLogoURL": "",
      "AdminName": "",
      "AdminEmail": "",
      "CustomData": "",
      "ContinentOnly": false,
      "CountryOnly": false,
      "ASOnly": false,
      "Score": 0,
      "Latitude": 0,
      "Longitude": 0,
      "ContinentCode": "NA",
      "CountryCodes": "US",
      "ExcludedCountryCodes": "",
      "Asnum": 0,
      "Comment": "",
      "Enabled": false,
      "StateSince": "0001-01-01T00:00:00Z",
      "Distance": 0,
      "ComputedScore": 0,
      "LastSync": "0001-01-01T00:00:00Z",
      "LastSuccessfulSync": "0001-01-01T00:00:00Z",
      "LastModTime": "0001-01-01T00:00:00Z"
    }
  ],
  "ExcludedList": [
    {
      "ID": 1,
      "Name": "westeurope",
      "HttpURL": "https://westeurope.cloudflare.jenkins.io/",
      "RsyncURL": "rsync://updates-jenkins-io-rsyncd.updates-jenkins-io.svc.cluster.local/jenkins/",
      "FtpURL": "",
      "SponsorName": "westeurope.cloudflare.jenkins.io",
      "SponsorURL": "",
      "SponsorLogoURL": "",
      "AdminName": "jenkins-infra",
      "AdminEmail": "[email protected]",
      "CustomData": "",
      "ContinentOnly": false,
      "CountryOnly": false,
      "ASOnly": false,
      "Score": 0,
      "Latitude": 48.8691,
      "Longitude": 2.324464,
      "ContinentCode": "EU",
      "CountryCodes": "FR",
      "ExcludedCountryCodes": "",
      "Asnum": 13335,
      "Comment": "",
      "Enabled": true,
      "ExcludeReason": "File size mismatch",
      "StateSince": "2024-09-11T19:27:57Z",
      "Distance": 280.6335,
      "ComputedScore": 0,
      "LastSync": "2024-09-11T19:56:56Z",
      "LastSuccessfulSync": "2024-09-11T19:56:56Z",
      "LastModTime": "2024-09-11T19:53:05Z"
    },
    {
      "ID": 2,
      "Name": "eastamerica",
      "HttpURL": "https://eastamerica.cloudflare.jenkins.io/",
      "RsyncURL": "rsync://updates-jenkins-io-rsyncd.updates-jenkins-io.svc.cluster.local/jenkins/",
      "FtpURL": "",
      "SponsorName": "eastamerica.cloudflare.jenkins.io",
      "SponsorURL": "",
      "SponsorLogoURL": "",
      "AdminName": "jenkins-infra",
      "AdminEmail": "[email protected]",
      "CustomData": "",
      "ContinentOnly": false,
      "CountryOnly": false,
      "ASOnly": false,
      "Score": 0,
      "Latitude": 35.221386,
      "Longitude": -80.83601,
      "ContinentCode": "NA",
      "CountryCodes": "US",
      "ExcludedCountryCodes": "",
      "Asnum": 13335,
      "Comment": "",
      "Enabled": true,
      "ExcludeReason": "File size mismatch",
      "StateSince": "2024-09-07T08:18:52Z",
      "Distance": 6445.7246,
      "ComputedScore": 0,
      "LastSync": "2024-09-11T19:57:10Z",
      "LastSuccessfulSync": "2024-09-11T19:57:10Z",
      "LastModTime": "2024-09-11T19:53:05Z"
    }
  ],
  "Fallback": true,
  "LocalJSPath": ""
}

timja · 2024-09-11T20:10:01Z

Reproducible in curl by passing the same headers (roughly, I had 2 drop a couple of compression algorithms as my curl couldn't handle whatever was being sent):

curl --compressed -H "User-Agent: axios/1.7.4" -H "Accept: application/json, text/plain, */*" -H "Accept-Encoding: gzip, deflate" -H "Host: mirrors.updates.jenkins.io" -H "Connection: keep-alive"  -L "https://updates.jenkins.io/plugin-versions.json"

timja · 2024-09-11T20:11:51Z

Minimised it down to an accept header, if you add an accept header its broken:

curl -H "Accept: application/json"  -L "https://updates.jenkins.io/plugin-versions.json"

I'll leave to you now @dduportal 😄

dduportal · 2024-09-12T06:01:31Z

Many thanks @timja !

Got the error, it's in mirrorbits. Incoming fix and I'll report in the Update Center helpdesk issue

dduportal · 2024-09-12T06:12:55Z

Reported in jenkins-infra/helpdesk#2649 (comment)

dduportal · 2024-09-12T07:48:07Z

All is green: self merging as hotfix

Continues from jenkins-infra#1890 Related with jenkins-infra/helpdesk#4265

Related with - jenkins-infra/helpdesk#4265 - jenkins-infra/plugin-site#1890

hotfix(utils.mjs) use the old stats website temporarily

734354c

dduportal changed the title ~~hotfix(utils.mjs) use the old stats website temporarily~~ hotfix(utils.mjs) use the old stats.jenkins.io website (temporarily?) Sep 11, 2024

dduportal marked this pull request as ready for review September 11, 2024 18:19

dduportal requested a review from a team as a code owner September 11, 2024 18:19

dduportal added 2 commits September 11, 2024 21:24

Update utils.mjs

89c4d7d

Update utils.mjs

5230f59

dduportal changed the title ~~hotfix(utils.mjs) use the old stats.jenkins.io website (temporarily?)~~ fix(utils.mjs) use the old stats.jenkins.io website (temporarily?) Sep 11, 2024

wip: Gavin suggestion

5489791

dduportal mentioned this pull request Sep 12, 2024

[INFRA-3100] Migrate updates.jenkins.io to another Cloud jenkins-infra/helpdesk#2649

Open

revert

2559526

infra-ci-jenkins-io bot had a problem deploying to preview September 12, 2024 06:42 Failure

infra-ci-jenkins-io bot deployed to preview September 12, 2024 07:05 View deployment

dduportal merged commit 2061e34 into main Sep 12, 2024
7 checks passed

dduportal deleted the dduportal-patch-1 branch September 12, 2024 07:48

krisstern mentioned this pull request Sep 12, 2024

Replacing existing stats.jenkins.io code with https://github.com/jenkins-infra/stats.jenkins.io jenkins-infra/helpdesk#4265

Closed

froque added a commit to froque/plugin-site that referenced this pull request Sep 12, 2024

use the old stats.jenkins.io website (temporarily?)

9acffe4

Continues from jenkins-infra#1890 Related with jenkins-infra/helpdesk#4265

froque mentioned this pull request Sep 12, 2024

use the old stats.jenkins.io website (temporarily?) #1893

Merged

froque added a commit to froque/shields that referenced this pull request Sep 12, 2024

Use old.stats.jenkins.io for JSON data

77f66e9

Related with - jenkins-infra/helpdesk#4265 - jenkins-infra/plugin-site#1890

froque mentioned this pull request Sep 12, 2024

[Jenkins] Use old.stats.jenkins.io for JSON data badges/shields#10522

Merged

dduportal mentioned this pull request Sep 12, 2024

To host stats.jenkins.io GSoC 2024 project in jenkins-infra jenkins-infra/helpdesk#4132

Closed

github-merge-queue bot pushed a commit to badges/shields that referenced this pull request Sep 13, 2024

Use old.stats.jenkins.io for JSON data (#10522)

394a827

Related with - jenkins-infra/helpdesk#4265 - jenkins-infra/plugin-site#1890

lemeurherve mentioned this pull request Oct 11, 2024

Revert "fix(utils.mjs) use the old stats.jenkins.io website (temporarily?)" #1934

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(utils.mjs) use the old stats.jenkins.io website (temporarily?) #1890

fix(utils.mjs) use the old stats.jenkins.io website (temporarily?) #1890

dduportal commented Sep 11, 2024 •

edited

Loading

dduportal commented Sep 11, 2024

dduportal commented Sep 11, 2024

dduportal commented Sep 11, 2024

timja commented Sep 11, 2024

halkeye commented Sep 11, 2024

dduportal commented Sep 11, 2024

dduportal commented Sep 11, 2024

timja commented Sep 11, 2024

timja commented Sep 11, 2024

timja commented Sep 11, 2024

timja commented Sep 11, 2024

dduportal commented Sep 12, 2024

dduportal commented Sep 12, 2024

dduportal commented Sep 12, 2024

fix(utils.mjs) use the old stats.jenkins.io website (temporarily?) #1890

fix(utils.mjs) use the old stats.jenkins.io website (temporarily?) #1890

Conversation

dduportal commented Sep 11, 2024 • edited Loading

dduportal commented Sep 11, 2024

dduportal commented Sep 11, 2024

dduportal commented Sep 11, 2024

timja commented Sep 11, 2024

halkeye commented Sep 11, 2024

dduportal commented Sep 11, 2024

dduportal commented Sep 11, 2024

timja commented Sep 11, 2024

timja commented Sep 11, 2024

timja commented Sep 11, 2024

timja commented Sep 11, 2024

dduportal commented Sep 12, 2024

dduportal commented Sep 12, 2024

dduportal commented Sep 12, 2024

dduportal commented Sep 11, 2024 •

edited

Loading